{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook records the experiments I have done in the article of \"Computing Semantic Similarity of Concepts in Knowledge Graphs\". If someone is interested in reproducing the experiments, one can install Sematch and use this notebook for reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sematch.semantic.similarity import WordNetSimilarity\n",
    "from IPython.display import display\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A simple example of word similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artist</th>\n",
       "      <th>musician</th>\n",
       "      <th>scientist</th>\n",
       "      <th>physicist</th>\n",
       "      <th>actor</th>\n",
       "      <th>movie</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>artist</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.809924</td>\n",
       "      <td>0.359417</td>\n",
       "      <td>0.296175</td>\n",
       "      <td>0.359417</td>\n",
       "      <td>0.135239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>musician</th>\n",
       "      <td>0.809924</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.296175</td>\n",
       "      <td>0.251859</td>\n",
       "      <td>0.641697</td>\n",
       "      <td>0.123384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>scientist</th>\n",
       "      <td>0.359417</td>\n",
       "      <td>0.296175</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.790743</td>\n",
       "      <td>0.456999</td>\n",
       "      <td>0.149615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>physicist</th>\n",
       "      <td>0.296175</td>\n",
       "      <td>0.251859</td>\n",
       "      <td>0.790743</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.359417</td>\n",
       "      <td>0.135239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actor</th>\n",
       "      <td>0.359417</td>\n",
       "      <td>0.641697</td>\n",
       "      <td>0.456999</td>\n",
       "      <td>0.359417</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.149615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie</th>\n",
       "      <td>0.135239</td>\n",
       "      <td>0.123384</td>\n",
       "      <td>0.149615</td>\n",
       "      <td>0.135239</td>\n",
       "      <td>0.149615</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             artist  musician  scientist  physicist     actor     movie\n",
       "artist     1.000000  0.809924   0.359417   0.296175  0.359417  0.135239\n",
       "musician   0.809924  1.000000   0.296175   0.251859  0.641697  0.123384\n",
       "scientist  0.359417  0.296175   1.000000   0.790743  0.456999  0.149615\n",
       "physicist  0.296175  0.251859   0.790743   1.000000  0.359417  0.135239\n",
       "actor      0.359417  0.641697   0.456999   0.359417  1.000000  0.149615\n",
       "movie      0.135239  0.123384   0.149615   0.135239  0.149615  1.000000"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wns = WordNetSimilarity()\n",
    "words = ['artist', 'musician', 'scientist', 'physicist', 'actor', 'movie']\n",
    "sim_matrix = [[wns.word_similarity(w1, w2, 'wpath') for w1 in words] for w2 in words]\n",
    "df = pd.DataFrame(sim_matrix, index=words,columns=words)\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluations on Word Similarity Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have collected some well known word similarity datasets for evaluating semantic similarity metrics. Several python classes can be used to separate the dataset for specicial purpose and evaluate the metric function automatically. \n",
    "\n",
    "We put them together and provide a uniformed framework to evaluate different semantic measures. The word similarity datasets include:\n",
    "\n",
    "- [Rubenstein and Goodenough (RG)](http://www.cs.cmu.edu/~mfaruqui/word-sim/EN-RG-65.txt) \n",
    "\n",
    "Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Commun. ACM 8, 10 (October 1965), 627-633. DOI=10.1145/365628.365657 \n",
    "\n",
    "- [Miller and Charles (MC)](http://www.cs.cmu.edu/~mfaruqui/word-sim/EN-MC-30.txt) \n",
    "\n",
    "Miller, George A., and Walter G. Charles. \"Contextual correlates of semantic similarity.\" Language and cognitive processes 6.1 (1991): 1-28.\n",
    "\n",
    "- [Wordsim353 (WS353)](http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/) \n",
    "\n",
    "Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin, \"Placing Search in Context: The Concept Revisited\", ACM Transactions on Information Systems, 20(1):116-131, January 2002 \n",
    "\n",
    "- [wordsim353 similarity and relatedness (WS353Sim)](http://alfonseca.org/eng/research/wordsim353.html) \n",
    "\n",
    "Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, Aitor Soroa, A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches, In Proceedings of NAACL-HLT 2009.\n",
    "\n",
    "- [SimLex-999 (SIMLEX)](http://www.cl.cam.ac.uk/~fh295/simlex.html) \n",
    "\n",
    "SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation. 2014. Felix Hill, Roi Reichart and Anna Korhonen. Preprint pubslished on arXiv. arXiv:1408.3456\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sematch.evaluation import WordSimEvaluation\n",
    "from sematch.semantic.similarity import WordNetSimilarity, YagoTypeSimilarity\n",
    "from nltk.corpus import wordnet as wn\n",
    "\n",
    "data_word_noun = ['noun_rg','noun_mc','noun_ws353','noun_ws353-sim','noun_simlex']\n",
    "data_word_graph = ['graph_rg','graph_mc','graph_ws353','graph_ws353-sim','graph_simlex']\n",
    "data_word_type = ['type_rg','type_mc','type_ws353','type_ws353-sim','type_simlex']\n",
    "\n",
    "sim_methods_noun = ['path','lch','wup','li','res','lin','jcn','wpath']\n",
    "sim_methods_graph = ['path','lch','wup','li','res','res_graph','lin','jcn','wpath','wpath_graph']\n",
    "sim_methods_type = ['path','lch','wup','li','res','res_graph','lin','lin_graph','jcn','jcn_graph','wpath','wpath_graph']\n",
    "\n",
    "ws_eval = WordSimEvaluation()\n",
    "wns = WordNetSimilarity()\n",
    "yagosim = YagoTypeSimilarity()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To produce the TABLE 2 in the article \"The illustration of Semantic Similarity Methods on Some Concept Pair Examples\". We manually create the word to synset mapping and compute their semantic similarity scores using different semantic similarity metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>path</th>\n",
       "      <th>lch</th>\n",
       "      <th>wup</th>\n",
       "      <th>li</th>\n",
       "      <th>res</th>\n",
       "      <th>lin</th>\n",
       "      <th>jcn</th>\n",
       "      <th>wpath</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>beef-octopus</th>\n",
       "      <td>0.200000</td>\n",
       "      <td>2.028148</td>\n",
       "      <td>0.714286</td>\n",
       "      <td>0.441994</td>\n",
       "      <td>6.109280</td>\n",
       "      <td>0.484051</td>\n",
       "      <td>0.071308</td>\n",
       "      <td>0.494238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beef-lamb</th>\n",
       "      <td>0.333333</td>\n",
       "      <td>2.538974</td>\n",
       "      <td>0.857143</td>\n",
       "      <td>0.667005</td>\n",
       "      <td>6.725370</td>\n",
       "      <td>0.591049</td>\n",
       "      <td>0.097025</td>\n",
       "      <td>0.691593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meat-seafood</th>\n",
       "      <td>0.333333</td>\n",
       "      <td>2.538974</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.659377</td>\n",
       "      <td>6.109280</td>\n",
       "      <td>0.759623</td>\n",
       "      <td>0.205488</td>\n",
       "      <td>0.661525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>octopus-shellfish</th>\n",
       "      <td>0.333333</td>\n",
       "      <td>2.538974</td>\n",
       "      <td>0.857143</td>\n",
       "      <td>0.667005</td>\n",
       "      <td>9.359654</td>\n",
       "      <td>0.729141</td>\n",
       "      <td>0.125726</td>\n",
       "      <td>0.801453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beef-service</th>\n",
       "      <td>0.071429</td>\n",
       "      <td>0.998529</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.049523</td>\n",
       "      <td>0.071429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beef-atmosphere</th>\n",
       "      <td>0.083333</td>\n",
       "      <td>1.152680</td>\n",
       "      <td>0.153846</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.052383</td>\n",
       "      <td>0.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beef-coffee</th>\n",
       "      <td>0.111111</td>\n",
       "      <td>1.440362</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>0.168312</td>\n",
       "      <td>3.336706</td>\n",
       "      <td>0.319125</td>\n",
       "      <td>0.065625</td>\n",
       "      <td>0.208354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>food-coffee</th>\n",
       "      <td>0.142857</td>\n",
       "      <td>1.691676</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.251092</td>\n",
       "      <td>3.336706</td>\n",
       "      <td>0.410819</td>\n",
       "      <td>0.094601</td>\n",
       "      <td>0.259764</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       path       lch       wup        li       res       lin  \\\n",
       "beef-octopus       0.200000  2.028148  0.714286  0.441994  6.109280  0.484051   \n",
       "beef-lamb          0.333333  2.538974  0.857143  0.667005  6.725370  0.591049   \n",
       "meat-seafood       0.333333  2.538974  0.833333  0.659377  6.109280  0.759623   \n",
       "octopus-shellfish  0.333333  2.538974  0.857143  0.667005  9.359654  0.729141   \n",
       "beef-service       0.071429  0.998529  0.133333  0.000000  0.000000  0.000000   \n",
       "beef-atmosphere    0.083333  1.152680  0.153846  0.000000  0.000000  0.000000   \n",
       "beef-coffee        0.111111  1.440362  0.428571  0.168312  3.336706  0.319125   \n",
       "food-coffee        0.142857  1.691676  0.500000  0.251092  3.336706  0.410819   \n",
       "\n",
       "                        jcn     wpath  \n",
       "beef-octopus       0.071308  0.494238  \n",
       "beef-lamb          0.097025  0.691593  \n",
       "meat-seafood       0.205488  0.661525  \n",
       "octopus-shellfish  0.125726  0.801453  \n",
       "beef-service       0.049523  0.071429  \n",
       "beef-atmosphere    0.052383  0.083333  \n",
       "beef-coffee        0.065625  0.208354  \n",
       "food-coffee        0.094601  0.259764  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "aspects = {'beef':wn.synset('beef.n.02'), 'lamb':wn.synset('lamb.n.05'), 'octopus':wn.synset('octopus.n.01'),\n",
    "          'shellfish':wn.synset('shellfish.n.01'), 'meat':wn.synset('meat.n.01'), 'seafood':wn.synset('seafood.n.01'),\n",
    "          'food':wn.synset('food.n.02'), 'service':wn.synset('service.n.02'),'atmosphere':wn.synset('atmosphere.n.01'),\n",
    "          'coffee':wn.synset('coffee.n.01')}\n",
    "aspect_pairs = [('beef', 'octopus'), ('beef', 'lamb'), ('meat','seafood'), ('octopus', 'shellfish'),\n",
    "               ('beef','service'),('beef','atmosphere'),('beef', 'coffee'), ('food','coffee')]\n",
    "aspects_sim_matrix = [[wns.similarity(aspects[w1], aspects[w2], m) for m in sim_methods_noun] \n",
    "                      for w1, w2 in aspect_pairs]\n",
    "aspect_index = [x+'-'+y for x, y in aspect_pairs]\n",
    "aspect_df = pd.DataFrame(aspects_sim_matrix, index=aspect_index, columns=sim_methods_noun)\n",
    "display(aspect_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### WPATH method with different K in Word Noun Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data_word_noun contains word pairs that can be mapped to WordNet noun taxonomy. The k settings are varied with interval 0.1 started from 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>noun_rg</th>\n",
       "      <th>noun_mc</th>\n",
       "      <th>noun_ws353</th>\n",
       "      <th>noun_ws353-sim</th>\n",
       "      <th>noun_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.1</th>\n",
       "      <td>0.747</td>\n",
       "      <td>0.703</td>\n",
       "      <td>0.278</td>\n",
       "      <td>0.535</td>\n",
       "      <td>0.486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>0.746</td>\n",
       "      <td>0.696</td>\n",
       "      <td>0.324</td>\n",
       "      <td>0.616</td>\n",
       "      <td>0.497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.3</th>\n",
       "      <td>0.776</td>\n",
       "      <td>0.737</td>\n",
       "      <td>0.343</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>0.785</td>\n",
       "      <td>0.740</td>\n",
       "      <td>0.347</td>\n",
       "      <td>0.643</td>\n",
       "      <td>0.573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.5</th>\n",
       "      <td>0.790</td>\n",
       "      <td>0.738</td>\n",
       "      <td>0.347</td>\n",
       "      <td>0.644</td>\n",
       "      <td>0.582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>0.789</td>\n",
       "      <td>0.732</td>\n",
       "      <td>0.345</td>\n",
       "      <td>0.642</td>\n",
       "      <td>0.589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.7</th>\n",
       "      <td>0.791</td>\n",
       "      <td>0.723</td>\n",
       "      <td>0.345</td>\n",
       "      <td>0.644</td>\n",
       "      <td>0.596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>0.794</td>\n",
       "      <td>0.728</td>\n",
       "      <td>0.341</td>\n",
       "      <td>0.645</td>\n",
       "      <td>0.603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.9</th>\n",
       "      <td>0.795</td>\n",
       "      <td>0.726</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.636</td>\n",
       "      <td>0.601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.781</td>\n",
       "      <td>0.724</td>\n",
       "      <td>0.310</td>\n",
       "      <td>0.609</td>\n",
       "      <td>0.584</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     noun_rg  noun_mc  noun_ws353  noun_ws353-sim  noun_simlex\n",
       "0.1    0.747    0.703       0.278           0.535        0.486\n",
       "0.2    0.746    0.696       0.324           0.616        0.497\n",
       "0.3    0.776    0.737       0.343           0.635        0.550\n",
       "0.4    0.785    0.740       0.347           0.643        0.573\n",
       "0.5    0.790    0.738       0.347           0.644        0.582\n",
       "0.6    0.789    0.732       0.345           0.642        0.589\n",
       "0.7    0.791    0.723       0.345           0.644        0.596\n",
       "0.8    0.794    0.728       0.341           0.645        0.603\n",
       "0.9    0.795    0.726       0.332           0.636        0.601\n",
       "1.0    0.781    0.724       0.310           0.609        0.584"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wpath_cors = [ws_eval.evaluate_wpath_k(dataset) for _, dataset in enumerate(data_word_noun)]\n",
    "cors_matrix = [[cors[i] for _, cors in enumerate(wpath_cors)] for i in range(1,11)]\n",
    "wpath_index = map(lambda x: str(x/10.0), range(1, 11))\n",
    "df_wpath = pd.DataFrame(cors_matrix, index=wpath_index, columns=data_word_noun)\n",
    "display(df_wpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### WPATH method with different K in Word Graph Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In word graph dataset, we performed the evaluation of wpath with different k using corpus-based IC and graph-based IC respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>graph_rg</th>\n",
       "      <th>graph_mc</th>\n",
       "      <th>graph_ws353</th>\n",
       "      <th>graph_ws353-sim</th>\n",
       "      <th>graph_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.1</th>\n",
       "      <td>0.734</td>\n",
       "      <td>0.673</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.512</td>\n",
       "      <td>0.479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>0.732</td>\n",
       "      <td>0.664</td>\n",
       "      <td>0.340</td>\n",
       "      <td>0.604</td>\n",
       "      <td>0.490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.3</th>\n",
       "      <td>0.769</td>\n",
       "      <td>0.711</td>\n",
       "      <td>0.361</td>\n",
       "      <td>0.625</td>\n",
       "      <td>0.546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>0.780</td>\n",
       "      <td>0.714</td>\n",
       "      <td>0.366</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.5</th>\n",
       "      <td>0.786</td>\n",
       "      <td>0.714</td>\n",
       "      <td>0.366</td>\n",
       "      <td>0.636</td>\n",
       "      <td>0.579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>0.786</td>\n",
       "      <td>0.708</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.7</th>\n",
       "      <td>0.789</td>\n",
       "      <td>0.696</td>\n",
       "      <td>0.367</td>\n",
       "      <td>0.636</td>\n",
       "      <td>0.593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>0.794</td>\n",
       "      <td>0.703</td>\n",
       "      <td>0.364</td>\n",
       "      <td>0.639</td>\n",
       "      <td>0.600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.9</th>\n",
       "      <td>0.796</td>\n",
       "      <td>0.700</td>\n",
       "      <td>0.354</td>\n",
       "      <td>0.631</td>\n",
       "      <td>0.599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.782</td>\n",
       "      <td>0.699</td>\n",
       "      <td>0.331</td>\n",
       "      <td>0.602</td>\n",
       "      <td>0.581</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     graph_rg  graph_mc  graph_ws353  graph_ws353-sim  graph_simlex\n",
       "0.1     0.734     0.673        0.286            0.512         0.479\n",
       "0.2     0.732     0.664        0.340            0.604         0.490\n",
       "0.3     0.769     0.711        0.361            0.625         0.546\n",
       "0.4     0.780     0.714        0.366            0.634         0.569\n",
       "0.5     0.786     0.714        0.366            0.636         0.579\n",
       "0.6     0.786     0.708        0.365            0.634         0.586\n",
       "0.7     0.789     0.696        0.367            0.636         0.593\n",
       "0.8     0.794     0.703        0.364            0.639         0.600\n",
       "0.9     0.796     0.700        0.354            0.631         0.599\n",
       "1.0     0.782     0.699        0.331            0.602         0.581"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#evaluate with corpus-based IC\n",
    "wpath_cors = [ws_eval.evaluate_wpath_k(dataset) for _, dataset in enumerate(data_word_graph)]\n",
    "cors_matrix = [[cors[i] for _, cors in enumerate(wpath_cors)] for i in range(1,11)]\n",
    "df_wpath_graph = pd.DataFrame(cors_matrix, index=wpath_index, columns=data_word_graph)\n",
    "display(df_wpath_graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>graph_rg</th>\n",
       "      <th>graph_mc</th>\n",
       "      <th>graph_ws353</th>\n",
       "      <th>graph_ws353-sim</th>\n",
       "      <th>graph_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.1</th>\n",
       "      <td>0.707</td>\n",
       "      <td>0.687</td>\n",
       "      <td>0.161</td>\n",
       "      <td>0.385</td>\n",
       "      <td>0.375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>0.729</td>\n",
       "      <td>0.722</td>\n",
       "      <td>0.249</td>\n",
       "      <td>0.474</td>\n",
       "      <td>0.404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.3</th>\n",
       "      <td>0.736</td>\n",
       "      <td>0.742</td>\n",
       "      <td>0.301</td>\n",
       "      <td>0.545</td>\n",
       "      <td>0.436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>0.757</td>\n",
       "      <td>0.757</td>\n",
       "      <td>0.313</td>\n",
       "      <td>0.566</td>\n",
       "      <td>0.468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.5</th>\n",
       "      <td>0.774</td>\n",
       "      <td>0.754</td>\n",
       "      <td>0.320</td>\n",
       "      <td>0.573</td>\n",
       "      <td>0.479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>0.781</td>\n",
       "      <td>0.761</td>\n",
       "      <td>0.327</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.7</th>\n",
       "      <td>0.781</td>\n",
       "      <td>0.725</td>\n",
       "      <td>0.330</td>\n",
       "      <td>0.591</td>\n",
       "      <td>0.519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>0.778</td>\n",
       "      <td>0.712</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.599</td>\n",
       "      <td>0.542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.9</th>\n",
       "      <td>0.794</td>\n",
       "      <td>0.716</td>\n",
       "      <td>0.339</td>\n",
       "      <td>0.616</td>\n",
       "      <td>0.571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.782</td>\n",
       "      <td>0.699</td>\n",
       "      <td>0.331</td>\n",
       "      <td>0.602</td>\n",
       "      <td>0.581</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     graph_rg  graph_mc  graph_ws353  graph_ws353-sim  graph_simlex\n",
       "0.1     0.707     0.687        0.161            0.385         0.375\n",
       "0.2     0.729     0.722        0.249            0.474         0.404\n",
       "0.3     0.736     0.742        0.301            0.545         0.436\n",
       "0.4     0.757     0.757        0.313            0.566         0.468\n",
       "0.5     0.774     0.754        0.320            0.573         0.479\n",
       "0.6     0.781     0.761        0.327            0.588         0.501\n",
       "0.7     0.781     0.725        0.330            0.591         0.519\n",
       "0.8     0.778     0.712        0.332            0.599         0.542\n",
       "0.9     0.794     0.716        0.339            0.616         0.571\n",
       "1.0     0.782     0.699        0.331            0.602         0.581"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#evaluate with graph-based IC\n",
    "wpath_cors = [ws_eval.evaluate_wpath_k(dataset, 'graph') for _, dataset in enumerate(data_word_graph)]\n",
    "cors_matrix = [[cors[i] for _, cors in enumerate(wpath_cors)] for i in range(1,11)]\n",
    "df_wpath_graph = pd.DataFrame(cors_matrix, index=wpath_index, columns=data_word_graph)\n",
    "display(df_wpath_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### WPATH method with different K in Word Type Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type_rg</th>\n",
       "      <th>type_mc</th>\n",
       "      <th>type_ws353</th>\n",
       "      <th>type_ws353-sim</th>\n",
       "      <th>type_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.1</th>\n",
       "      <td>0.632</td>\n",
       "      <td>0.608</td>\n",
       "      <td>0.280</td>\n",
       "      <td>0.478</td>\n",
       "      <td>0.482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>0.632</td>\n",
       "      <td>0.584</td>\n",
       "      <td>0.327</td>\n",
       "      <td>0.563</td>\n",
       "      <td>0.503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.3</th>\n",
       "      <td>0.672</td>\n",
       "      <td>0.659</td>\n",
       "      <td>0.357</td>\n",
       "      <td>0.595</td>\n",
       "      <td>0.556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>0.683</td>\n",
       "      <td>0.646</td>\n",
       "      <td>0.361</td>\n",
       "      <td>0.600</td>\n",
       "      <td>0.582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.5</th>\n",
       "      <td>0.689</td>\n",
       "      <td>0.669</td>\n",
       "      <td>0.363</td>\n",
       "      <td>0.602</td>\n",
       "      <td>0.594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>0.688</td>\n",
       "      <td>0.667</td>\n",
       "      <td>0.363</td>\n",
       "      <td>0.598</td>\n",
       "      <td>0.603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.7</th>\n",
       "      <td>0.688</td>\n",
       "      <td>0.638</td>\n",
       "      <td>0.367</td>\n",
       "      <td>0.602</td>\n",
       "      <td>0.613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>0.691</td>\n",
       "      <td>0.627</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.606</td>\n",
       "      <td>0.621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.9</th>\n",
       "      <td>0.689</td>\n",
       "      <td>0.620</td>\n",
       "      <td>0.359</td>\n",
       "      <td>0.606</td>\n",
       "      <td>0.625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.679</td>\n",
       "      <td>0.621</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.601</td>\n",
       "      <td>0.616</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     type_rg  type_mc  type_ws353  type_ws353-sim  type_simlex\n",
       "0.1    0.632    0.608       0.280           0.478        0.482\n",
       "0.2    0.632    0.584       0.327           0.563        0.503\n",
       "0.3    0.672    0.659       0.357           0.595        0.556\n",
       "0.4    0.683    0.646       0.361           0.600        0.582\n",
       "0.5    0.689    0.669       0.363           0.602        0.594\n",
       "0.6    0.688    0.667       0.363           0.598        0.603\n",
       "0.7    0.688    0.638       0.367           0.602        0.613\n",
       "0.8    0.691    0.627       0.365           0.606        0.621\n",
       "0.9    0.689    0.620       0.359           0.606        0.625\n",
       "1.0    0.679    0.621       0.353           0.601        0.616"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#evaluate with corpus-based IC\n",
    "wpath_cors = [ws_eval.evaluate_wpath_k(dataset) for _, dataset in enumerate(data_word_type)]\n",
    "cors_matrix = [[cors[i] for _, cors in enumerate(wpath_cors)] for i in range(1,11)]\n",
    "df_wpath_type = pd.DataFrame(cors_matrix, index=wpath_index, columns=data_word_type)\n",
    "display(df_wpath_type)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type_rg</th>\n",
       "      <th>type_mc</th>\n",
       "      <th>type_ws353</th>\n",
       "      <th>type_ws353-sim</th>\n",
       "      <th>type_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.1</th>\n",
       "      <td>0.653</td>\n",
       "      <td>0.670</td>\n",
       "      <td>0.175</td>\n",
       "      <td>0.398</td>\n",
       "      <td>0.387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>0.692</td>\n",
       "      <td>0.726</td>\n",
       "      <td>0.248</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.3</th>\n",
       "      <td>0.696</td>\n",
       "      <td>0.729</td>\n",
       "      <td>0.304</td>\n",
       "      <td>0.504</td>\n",
       "      <td>0.477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>0.698</td>\n",
       "      <td>0.729</td>\n",
       "      <td>0.308</td>\n",
       "      <td>0.521</td>\n",
       "      <td>0.512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.5</th>\n",
       "      <td>0.713</td>\n",
       "      <td>0.759</td>\n",
       "      <td>0.318</td>\n",
       "      <td>0.529</td>\n",
       "      <td>0.527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>0.714</td>\n",
       "      <td>0.766</td>\n",
       "      <td>0.327</td>\n",
       "      <td>0.546</td>\n",
       "      <td>0.550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.7</th>\n",
       "      <td>0.703</td>\n",
       "      <td>0.732</td>\n",
       "      <td>0.331</td>\n",
       "      <td>0.553</td>\n",
       "      <td>0.567</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>0.687</td>\n",
       "      <td>0.709</td>\n",
       "      <td>0.334</td>\n",
       "      <td>0.565</td>\n",
       "      <td>0.589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.9</th>\n",
       "      <td>0.692</td>\n",
       "      <td>0.656</td>\n",
       "      <td>0.343</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.679</td>\n",
       "      <td>0.621</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.601</td>\n",
       "      <td>0.616</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     type_rg  type_mc  type_ws353  type_ws353-sim  type_simlex\n",
       "0.1    0.653    0.670       0.175           0.398        0.387\n",
       "0.2    0.692    0.726       0.248           0.436        0.431\n",
       "0.3    0.696    0.729       0.304           0.504        0.477\n",
       "0.4    0.698    0.729       0.308           0.521        0.512\n",
       "0.5    0.713    0.759       0.318           0.529        0.527\n",
       "0.6    0.714    0.766       0.327           0.546        0.550\n",
       "0.7    0.703    0.732       0.331           0.553        0.567\n",
       "0.8    0.687    0.709       0.334           0.565        0.589\n",
       "0.9    0.692    0.656       0.343           0.585        0.611\n",
       "1.0    0.679    0.621       0.353           0.601        0.616"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#evaluate with graph-based IC\n",
    "wpath_cors = [ws_eval.evaluate_wpath_k(dataset, 'graph') for _, dataset in enumerate(data_word_type)]\n",
    "cors_matrix = [[cors[i] for _, cors in enumerate(wpath_cors)] for i in range(1,11)]\n",
    "df_wpath_type = pd.DataFrame(cors_matrix, index=wpath_index, columns=data_word_type)\n",
    "display(df_wpath_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Baseline semantic similarity metrics on Word Noun Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>noun_rg</th>\n",
       "      <th>noun_mc</th>\n",
       "      <th>noun_ws353</th>\n",
       "      <th>noun_ws353-sim</th>\n",
       "      <th>noun_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>path</th>\n",
       "      <td>0.781</td>\n",
       "      <td>0.724</td>\n",
       "      <td>0.310</td>\n",
       "      <td>0.609</td>\n",
       "      <td>0.584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lch</th>\n",
       "      <td>0.781</td>\n",
       "      <td>0.724</td>\n",
       "      <td>0.310</td>\n",
       "      <td>0.609</td>\n",
       "      <td>0.584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wup</th>\n",
       "      <td>0.755</td>\n",
       "      <td>0.729</td>\n",
       "      <td>0.344</td>\n",
       "      <td>0.624</td>\n",
       "      <td>0.542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>li</th>\n",
       "      <td>0.787</td>\n",
       "      <td>0.719</td>\n",
       "      <td>0.334</td>\n",
       "      <td>0.628</td>\n",
       "      <td>0.586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>res</th>\n",
       "      <td>0.776</td>\n",
       "      <td>0.733</td>\n",
       "      <td>0.346</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lin</th>\n",
       "      <td>0.784</td>\n",
       "      <td>0.752</td>\n",
       "      <td>0.307</td>\n",
       "      <td>0.603</td>\n",
       "      <td>0.582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jcn</th>\n",
       "      <td>0.775</td>\n",
       "      <td>0.820</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.581</td>\n",
       "      <td>0.579</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      noun_rg  noun_mc  noun_ws353  noun_ws353-sim  noun_simlex\n",
       "path    0.781    0.724       0.310           0.609        0.584\n",
       "lch     0.781    0.724       0.310           0.609        0.584\n",
       "wup     0.755    0.729       0.344           0.624        0.542\n",
       "li      0.787    0.719       0.334           0.628        0.586\n",
       "res     0.776    0.733       0.346           0.634        0.535\n",
       "lin     0.784    0.752       0.307           0.603        0.582\n",
       "jcn     0.775    0.820       0.284           0.581        0.579"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "path = lambda x, y: wns.word_similarity(x, y, 'path')\n",
    "lch = lambda x, y: wns.word_similarity(x, y, 'lch')\n",
    "wup = lambda x, y: wns.word_similarity(x, y, 'wup')\n",
    "li = lambda x, y: wns.word_similarity(x, y, 'li')\n",
    "res = lambda x, y: wns.word_similarity(x, y, 'res')\n",
    "lin = lambda x, y: wns.word_similarity(x, y, 'lin')\n",
    "jcn = lambda x, y: wns.word_similarity(x, y, 'jcn')\n",
    "\n",
    "methods = {'path':path, 'lch':lch, 'wup':wup, 'li':li, 'res':res, 'lin':lin, 'jcn':jcn}\n",
    "cor_dicts = [ws_eval.evaluate_multiple_metrics(methods, dataset) for dataset in data_word_noun]\n",
    "baseline_cors_matrix = [[cors[m] for _, cors in enumerate(cor_dicts)] for m in sim_methods_noun[0:7]]\n",
    "df_baselines_noun = pd.DataFrame(baseline_cors_matrix, index=sim_methods_noun[0:7], columns=data_word_noun)\n",
    "display(df_baselines_noun)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Baseline semantic similarity metrics on Word Graph Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>graph_rg</th>\n",
       "      <th>graph_mc</th>\n",
       "      <th>graph_ws353</th>\n",
       "      <th>graph_ws353-sim</th>\n",
       "      <th>graph_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>path</th>\n",
       "      <td>0.782</td>\n",
       "      <td>0.699</td>\n",
       "      <td>0.331</td>\n",
       "      <td>0.602</td>\n",
       "      <td>0.581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lch</th>\n",
       "      <td>0.782</td>\n",
       "      <td>0.699</td>\n",
       "      <td>0.331</td>\n",
       "      <td>0.602</td>\n",
       "      <td>0.581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wup</th>\n",
       "      <td>0.738</td>\n",
       "      <td>0.711</td>\n",
       "      <td>0.362</td>\n",
       "      <td>0.612</td>\n",
       "      <td>0.537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>li</th>\n",
       "      <td>0.779</td>\n",
       "      <td>0.696</td>\n",
       "      <td>0.349</td>\n",
       "      <td>0.615</td>\n",
       "      <td>0.583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>res</th>\n",
       "      <td>0.765</td>\n",
       "      <td>0.713</td>\n",
       "      <td>0.363</td>\n",
       "      <td>0.622</td>\n",
       "      <td>0.530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>res_graph</th>\n",
       "      <td>0.723</td>\n",
       "      <td>0.717</td>\n",
       "      <td>0.310</td>\n",
       "      <td>0.540</td>\n",
       "      <td>0.366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lin</th>\n",
       "      <td>0.776</td>\n",
       "      <td>0.736</td>\n",
       "      <td>0.320</td>\n",
       "      <td>0.589</td>\n",
       "      <td>0.578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jcn</th>\n",
       "      <td>0.762</td>\n",
       "      <td>0.794</td>\n",
       "      <td>0.301</td>\n",
       "      <td>0.578</td>\n",
       "      <td>0.576</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           graph_rg  graph_mc  graph_ws353  graph_ws353-sim  graph_simlex\n",
       "path          0.782     0.699        0.331            0.602         0.581\n",
       "lch           0.782     0.699        0.331            0.602         0.581\n",
       "wup           0.738     0.711        0.362            0.612         0.537\n",
       "li            0.779     0.696        0.349            0.615         0.583\n",
       "res           0.765     0.713        0.363            0.622         0.530\n",
       "res_graph     0.723     0.717        0.310            0.540         0.366\n",
       "lin           0.776     0.736        0.320            0.589         0.578\n",
       "jcn           0.762     0.794        0.301            0.578         0.576"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res_graph = lambda x, y: yagosim.word_similarity(x, y, 'res_graph')\n",
    "methods['res_graph'] = res_graph\n",
    "cor_dicts = [ws_eval.evaluate_multiple_metrics(methods, dataset) for dataset in data_word_graph]\n",
    "baseline_cors_matrix = [[cors[m] for _, cors in enumerate(cor_dicts)] for m in sim_methods_graph[0:8]]\n",
    "df_baselines_graph = pd.DataFrame(baseline_cors_matrix, index=sim_methods_graph[0:8], columns=data_word_graph)\n",
    "display(df_baselines_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Baseline semantic similarity metrics on Word Type Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type_rg</th>\n",
       "      <th>type_mc</th>\n",
       "      <th>type_ws353</th>\n",
       "      <th>type_ws353-sim</th>\n",
       "      <th>type_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>path</th>\n",
       "      <td>0.679</td>\n",
       "      <td>0.621</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.601</td>\n",
       "      <td>0.616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lch</th>\n",
       "      <td>0.679</td>\n",
       "      <td>0.621</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.601</td>\n",
       "      <td>0.616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wup</th>\n",
       "      <td>0.613</td>\n",
       "      <td>0.606</td>\n",
       "      <td>0.357</td>\n",
       "      <td>0.589</td>\n",
       "      <td>0.538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>li</th>\n",
       "      <td>0.673</td>\n",
       "      <td>0.614</td>\n",
       "      <td>0.361</td>\n",
       "      <td>0.612</td>\n",
       "      <td>0.612</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>res</th>\n",
       "      <td>0.667</td>\n",
       "      <td>0.679</td>\n",
       "      <td>0.355</td>\n",
       "      <td>0.595</td>\n",
       "      <td>0.540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>res_graph</th>\n",
       "      <td>0.676</td>\n",
       "      <td>0.704</td>\n",
       "      <td>0.294</td>\n",
       "      <td>0.490</td>\n",
       "      <td>0.387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lin</th>\n",
       "      <td>0.642</td>\n",
       "      <td>0.696</td>\n",
       "      <td>0.321</td>\n",
       "      <td>0.539</td>\n",
       "      <td>0.592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lin_graph</th>\n",
       "      <td>0.627</td>\n",
       "      <td>0.661</td>\n",
       "      <td>0.324</td>\n",
       "      <td>0.547</td>\n",
       "      <td>0.527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jcn</th>\n",
       "      <td>0.676</td>\n",
       "      <td>0.805</td>\n",
       "      <td>0.340</td>\n",
       "      <td>0.546</td>\n",
       "      <td>0.594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jcn_graph</th>\n",
       "      <td>-0.367</td>\n",
       "      <td>-0.324</td>\n",
       "      <td>-0.211</td>\n",
       "      <td>-0.372</td>\n",
       "      <td>-0.334</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           type_rg  type_mc  type_ws353  type_ws353-sim  type_simlex\n",
       "path         0.679    0.621       0.353           0.601        0.616\n",
       "lch          0.679    0.621       0.353           0.601        0.616\n",
       "wup          0.613    0.606       0.357           0.589        0.538\n",
       "li           0.673    0.614       0.361           0.612        0.612\n",
       "res          0.667    0.679       0.355           0.595        0.540\n",
       "res_graph    0.676    0.704       0.294           0.490        0.387\n",
       "lin          0.642    0.696       0.321           0.539        0.592\n",
       "lin_graph    0.627    0.661       0.324           0.547        0.527\n",
       "jcn          0.676    0.805       0.340           0.546        0.594\n",
       "jcn_graph   -0.367   -0.324      -0.211          -0.372       -0.334"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lin_graph = lambda x, y: yagosim.word_similarity(x, y, 'lin_graph')\n",
    "jcn_graph = lambda x, y: yagosim.word_similarity(x, y, 'jcn_graph')\n",
    "methods['lin_graph'] = lin_graph\n",
    "methods['jcn_graph'] = jcn_graph\n",
    "cor_dicts = [ws_eval.evaluate_multiple_metrics(methods, dataset) for dataset in data_word_type]\n",
    "baseline_cors_matrix = [[cors[m] for _, cors in enumerate(cor_dicts)] for m in sim_methods_type[0:10]]\n",
    "df_baselines_type = pd.DataFrame(baseline_cors_matrix, index=sim_methods_type[0:10], columns=data_word_type)\n",
    "display(df_baselines_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Steiger's Z Significance Test on Word Noun Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "wpath_rg = lambda x, y: wns.word_similarity_wpath(x, y, 0.9)\n",
    "wpath_mc = lambda x, y: wns.word_similarity_wpath(x, y, 0.4)\n",
    "wpath_ws353 = lambda x, y: wns.word_similarity_wpath(x, y, 0.5)\n",
    "wpath_ws353sim = lambda x, y: wns.word_similarity_wpath(x, y, 0.8)\n",
    "wpath_simlex = lambda x, y: wns.word_similarity_wpath(x, y, 0.8)\n",
    "\n",
    "methods = {'wpath_rg':wpath_rg, 'wpath_mc':wpath_mc, 'wpath_ws353':wpath_ws353, \n",
    "           'wpath_ws353sim':wpath_ws353sim,'wpath_simlex':wpath_simlex}\n",
    "\n",
    "cor_dicts = [ws_eval.evaluate_multiple_metrics(methods, dataset) for dataset in data_word_noun]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>noun_rg</th>\n",
       "      <th>noun_mc</th>\n",
       "      <th>noun_ws353</th>\n",
       "      <th>noun_ws353-sim</th>\n",
       "      <th>noun_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>metrics</th>\n",
       "      <td>0.795</td>\n",
       "      <td>0.74</td>\n",
       "      <td>0.347</td>\n",
       "      <td>0.645</td>\n",
       "      <td>0.603</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         noun_rg  noun_mc  noun_ws353  noun_ws353-sim  noun_simlex\n",
       "metrics    0.795     0.74       0.347           0.645        0.603"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wpath_dic = {'noun_rg':'wpath_rg', 'noun_mc':'wpath_mc', 'noun_ws353':'wpath_ws353',\n",
    "            'noun_ws353-sim':'wpath_ws353sim', 'noun_simlex':'wpath_simlex'}\n",
    "\n",
    "cors_matrix = [[cor_dicts[i][wpath_dic[dataset]] for i, dataset in enumerate(data_word_noun)]]\n",
    "df_cors = pd.DataFrame(cors_matrix, index=['metrics'], columns=data_word_noun)\n",
    "display(df_cors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform the Steiger's Z Significance Test, one can use the implementation integrated in Sematch framework, or use the R, cocor package. The example scripts using cocor package to perform statistical test in Simlex dataset is shown as:\n",
    "```\n",
    "require(cocor) # load package\n",
    "#j means dependent sample, k and h means comparison sample\n",
    "#we have wpath with human (jk), jcn with human (jh), and wpath with jcn (kh)\n",
    "#simlex\n",
    "#wpath with path Pass\n",
    "cocor.dep.groups.overlap(r.jk=+0.603, r.jh=+0.584, r.kh=+0.955, n=666, alternative=\"greater\", alpha=0.05, conf.level=0.95, null.value=0)\n",
    "#wpath with lch Pass\n",
    "cocor.dep.groups.overlap(r.jk=+0.603, r.jh=+0.584, r.kh=+0.955, n=666, alternative=\"greater\", alpha=0.05, conf.level=0.95, null.value=0)\n",
    "#wpath with wup Pass\n",
    "cocor.dep.groups.overlap(r.jk=+0.603, r.jh=+0.542, r.kh=+0.946, n=666, alternative=\"greater\", alpha=0.05, conf.level=0.95, null.value=0)\n",
    "#wpath with li Pass\n",
    " cocor.dep.groups.overlap(r.jk=+0.603, r.jh=+0.586, r.kh=+0.965, n=666, alternative=\"greater\", alpha=0.05, conf.level=0.95, null.value=0)\n",
    "#wpath with res Pass\n",
    "cocor.dep.groups.overlap(r.jk=+0.603, r.jh=+0.535, r.kh=+0.913, n=666, alternative=\"greater\", alpha=0.05, conf.level=0.95, null.value=0)\n",
    "#wpath with lin Pass\n",
    "cocor.dep.groups.overlap(r.jk=+0.603, r.jh=+0.582, r.kh=+0.944, n=666, alternative=\"greater\", alpha=0.05, conf.level=0.95, null.value=0)\n",
    "```\n",
    "The example of using the integrated Statistical Test is illustrate in the following codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>noun_rg</th>\n",
       "      <th>noun_mc</th>\n",
       "      <th>noun_ws353</th>\n",
       "      <th>noun_ws353-sim</th>\n",
       "      <th>noun_simlex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>path</th>\n",
       "      <td>(0.988,0.12237761609)</td>\n",
       "      <td>(0.948,0.351528419054)</td>\n",
       "      <td>(0.922,0.0321679269393)</td>\n",
       "      <td>(0.959,0.0107682312797)</td>\n",
       "      <td>(0.955,0.0206179473294)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lch</th>\n",
       "      <td>(0.988,0.12237761609)</td>\n",
       "      <td>(0.948,0.351528419054)</td>\n",
       "      <td>(0.922,0.0321679269393)</td>\n",
       "      <td>(0.959,0.0107682312797)</td>\n",
       "      <td>(0.955,0.0206179473294)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wup</th>\n",
       "      <td>(0.956,0.0425083374454)</td>\n",
       "      <td>(0.931,0.409258605134)</td>\n",
       "      <td>(0.956,0.420612449275)</td>\n",
       "      <td>(0.958,0.0917875911784)</td>\n",
       "      <td>(0.946,1.38934863614e-09)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>li</th>\n",
       "      <td>(0.984,0.281665699419)</td>\n",
       "      <td>(0.948,0.308981784008)</td>\n",
       "      <td>(0.95,0.208040059233)</td>\n",
       "      <td>(0.973,0.0897631401424)</td>\n",
       "      <td>(0.965,0.0192252476075)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>res</th>\n",
       "      <td>(0.939,0.239356192156)</td>\n",
       "      <td>(0.972,0.410158670791)</td>\n",
       "      <td>(0.984,0.455938513267)</td>\n",
       "      <td>(0.948,0.264502694351)</td>\n",
       "      <td>(0.913,9.47692808673e-08)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lin</th>\n",
       "      <td>(0.942,0.336047605166)</td>\n",
       "      <td>(0.976,0.334545281366)</td>\n",
       "      <td>(0.916,0.0270064898595)</td>\n",
       "      <td>(0.899,0.0431967198107)</td>\n",
       "      <td>(0.944,0.0215151648688)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jcn</th>\n",
       "      <td>(0.866,0.302109070882)</td>\n",
       "      <td>(0.901,0.057509130222)</td>\n",
       "      <td>(0.818,0.0197407219215)</td>\n",
       "      <td>(0.843,0.0181782667967)</td>\n",
       "      <td>(0.916,0.0292250544495)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      noun_rg                 noun_mc  \\\n",
       "path    (0.988,0.12237761609)  (0.948,0.351528419054)   \n",
       "lch     (0.988,0.12237761609)  (0.948,0.351528419054)   \n",
       "wup   (0.956,0.0425083374454)  (0.931,0.409258605134)   \n",
       "li     (0.984,0.281665699419)  (0.948,0.308981784008)   \n",
       "res    (0.939,0.239356192156)  (0.972,0.410158670791)   \n",
       "lin    (0.942,0.336047605166)  (0.976,0.334545281366)   \n",
       "jcn    (0.866,0.302109070882)  (0.901,0.057509130222)   \n",
       "\n",
       "                   noun_ws353           noun_ws353-sim  \\\n",
       "path  (0.922,0.0321679269393)  (0.959,0.0107682312797)   \n",
       "lch   (0.922,0.0321679269393)  (0.959,0.0107682312797)   \n",
       "wup    (0.956,0.420612449275)  (0.958,0.0917875911784)   \n",
       "li      (0.95,0.208040059233)  (0.973,0.0897631401424)   \n",
       "res    (0.984,0.455938513267)   (0.948,0.264502694351)   \n",
       "lin   (0.916,0.0270064898595)  (0.899,0.0431967198107)   \n",
       "jcn   (0.818,0.0197407219215)  (0.843,0.0181782667967)   \n",
       "\n",
       "                    noun_simlex  \n",
       "path    (0.955,0.0206179473294)  \n",
       "lch     (0.955,0.0206179473294)  \n",
       "wup   (0.946,1.38934863614e-09)  \n",
       "li      (0.965,0.0192252476075)  \n",
       "res   (0.913,9.47692808673e-08)  \n",
       "lin     (0.944,0.0215151648688)  \n",
       "jcn     (0.916,0.0292250544495)  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "stats_tests = []\n",
    "for _, dataset in enumerate(data_word_noun):\n",
    "    stats = {}\n",
    "    for _, m in enumerate(sim_methods_noun[0:7]):\n",
    "        cor, p_value = ws_eval.statistical_test(wpath_dic[dataset], m, dataset)\n",
    "        stats[m] = '('+str(round(cor,3))+','+str(p_value)+')'\n",
    "    stats_tests.append(stats)\n",
    "stats_matrix = [[cors[m] for _, cors in enumerate(stats_tests)] for _, m in enumerate(sim_methods_noun[0:7])]\n",
    "df_stats = pd.DataFrame(stats_matrix, index=sim_methods_noun[0:7], columns=data_word_noun)\n",
    "display(df_stats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
